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DETAILED ACTION 
Continued Examination Under 37 CFR 1.114 
1 . A request for continued examination under 37 CFR 1.114, including the fee set forth in 
37 CFR 1 .17(e), was filed in this application after final rejection. Since this application is 
eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) 
has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 
37 CFR 1.114. Applicant's submission filed on 8/3/2006 has been entered. 

Claim Rejections - 35 USC § 103 

1 . The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

Claims 1-6, and 12 are rejected under 35 U.S.C. 103(a) as being unpatentable over Guell et al. 
(NPL Document "FLILO - an Enhanced Vision System", herein referred to as "Guell") in view 
of NASA et al. (NPL Document "NASA's High-Speed Research Program", herein referred to as 
"NASA") in further view of Boorman et al. (US Pub 2004/0059474, herein referred to as 
"Boorman"). 

As per claim 1, Guell teaches the claimed "first sensor" and teaches the claimed "second 
sensor" by teaching of a plurality of imaging sensors shown on pg. 34, figure 4. 

Guell teaches the claimed "fuse the images to a single viewpoint" by teaching of "The 
video images from the cameras are digitized, and electronically combined in the Electronic 
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Imaging System (EIS) Processor to form an external view to the pilot" (pg. 33, 1st full paragraph 
in 1st col). 

Guell teaches the claimed "transform the fused image to a first viewpoint image ... and a 
second viewpoint" by teaching of in figure 6 where a cockpit is shown with displayed titled, 
images. If a pilot and co-pilot were both sitting in the cockpit, each pilot (operator) would have a 
different viewpoint image from their respective positions within the cockpit where the set of tiled 
images in constructed from fused images from multiple sensors. Further, Guell teaches the use 
of having multiple pilots in the cockpit by teaching of "FLILO would help retain the two-man 
cockpit crew" (pg. 34, last of line of 1st col and first line of 2nd col). Further, Guell teaches the 
claimed limitation by teaching of "Each pilot has an independent view, which is controlled by 
their head position, while utilizing the same sensors that are static and fixed to the aircraft" (pg. 
32, middle of 1st col). 

Guell does not explicitly teach the claimed "detect moving objects in the images". 
NASA teaches the claimed limitation by teaching of a feature described as "Object Detection: 
Data was gathered to help develop methods for detecting other aircraft" (third item in bulleted 
list under section "XVS Flight Test Series II"). 

It would have been obvious to one of ordinary skill in the art at the time of invention to 
combine Guell and NASA. One advantage of the combination is provided by NASA, which 
teaches of "In addition to providing valuable real-time sensor data for subsequent analyses, the 
first XVS flight test series gave researches confidence that a future supersonic passenger jet 
could indeed be flown without forward facing windows in the cockpit" (3rd paragraph under the 
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section "XVS Flight Test Series I") where the functionality of moving object detection would aid 
in achieving a windowless cockpit. 

Guell does not explicitly teach the claimed "generate a common display area associated 
with at least two mutually exclusive windows of information on the display device, wherein the 
common display area includes selectable options to allow the operator to display detailed 
information related to the information displayed in at least one of in the associated windows". 
Boorman teaches the claimed limitation by teaching of in figure 5, common display area 326, 
with at least two mutually exclusive windows 570, 580, and 590 with selectable options (i.e. 
574) to allow the operator to display the detailed information (i.e. the flight data as seen in the 
figure). 

It would have been obvious to one of ordinary skill in the art at the time of invention to 
combine Boorman with the combinable system of Guell and NASA. One advantage of the 
combination is provided by Boorman, which teaches of providing most of the current autoflight 
guidance instructions in one central location rather than the pilot looking in many different 
directions (paragraphs [0005] and [0006]). 

As per claim 2, Guell does not explicitly teach the claimed "combine the first and second 
viewpoint images with symbols". NASA teaches the claimed limitation by teaching of "pilot's 
ability to control and land the aircraft relying only on sensors and computer-generated images 
(including various symbols) on the XVS display" (2nd paragraph under the section titled "XVS 
Flight Test Series I"). It would have been obvious to one of ordinary skill in the art to use the 
claimed feature in order to provide useful data to the pilot when navigating an aircraft. 
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As per claim 3, Guell does not explicitly teach the claimed "detecting moving objects ... 
are to configured to execute simultaneously". NASA teaches the claimed limitation by teaching 
of multiple sensors (third paragraph under section "XVS Flight Test Series II") where these 
sensors would both be utilized to detect moving objects. It would have been obvious to one of 
ordinary skill in the art to use the claimed feature in order to provide useful and important data 
relating to moving aircraft to the pilot using the visualization system. 

As per claim 4, Guell teaches the claimed "transforming the fused image to the first 
viewpoint image" and teaches the claimed "transforming the fused image to the second 
viewpoint image" by teaching of on pg. 32, figure 1, where the plurality of imaging sensors data 
is inputted into an "Electronic Imaging System (EIS) processor" and the resulting image data is 
then sent to a pilot for display. While Guell does not explicitly teach the claimed use of a 
plurality of processors, it would have been obvious to use a plurality of processors for real-time 
data processing in order to send the image data to more than one operator (different viewpoints) 
because it is well known in the art that multi-processor use can be much faster. 

As per claim 5, Guell does not explicitly teach the claimed "symbols represent the 
moving objects in the vicinity of the device". NASA teaches the claimed limitation by teaching 
of computer-generated symbols on the display (2 nd paragraph under section titled 'XVS Flight 
Test Series I") and by teaching of detecting moving objects (third item in bulleted list under 
section "XVS Flight Test Series II"). It would have been obvious to one of ordinary skill in the 
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art to use the claimed feature in order to provide convenient and useful data to the pilot when 
navigating an aircraft. 

As per claim 6, Guell does not teach the claimed limitation. Boorman teaches the 
claimed environmental information in figure 5 where information about the aircraft's 
environment is shown in 570, 580, and 590. It would have been obvious to one of ordinary skill 
in the art to use the claimed feature in order to provide useful and important environmental data 
to the pilot when navigating an aircraft. 

As per claim 12, Guell teaches the claimed "wherein the first sensor and the second 
sensor are video cameras" by teaching of "video images from the cameras are digitized, and 
electronically combined in the Electronic Imaging System (EIS)" (pg. 33, first full paragraph in 
first column). 

2. Claims 7-11, 13, and 14 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Guell in view of NASA in further view of Boorman in further view of Bernier et al. (US Pub 
2004/0169663, herein referred to as "Bernier"). 

As per claim 7, Guell does not explicitly teach the claimed "wherein the symbols 
represent weather hazards in the vicinity of the device". NASA teaches of computer-generated 
symbols on the display (2 nd paragraph under section titled 'XVS Flight Test Series I"). Bernier 
teaches the claimed weather hazards "Visual hindrances may be due to bad weather, such as fog, 
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snow, or rain, or they may be due to the time of day, such as night, dawn, or dusk. Further, some 
visual hindrances are due to the field of view limitations of the aircraft itself. Many aircraft 
cockpits have a field of view that is typically limited to a forward facing area that does not 
provide the pilot with adequate visualization to the sides and rear of the aircraft" (paragraph 
[0005]). 

It would have been obvious to one of ordinary skill in the art at the time of invention to 
combine Bernier with the combinable system of Guell, NASA, and Boorman. Bernier teaches 
one advantage to the combining visible and computer generated image features by teaching of 
"The images are then summed together in this intensity percentage ratio, thereby providing the 
user with the benefit of both images" (paragraph [0017]). 

As per claim 8, Guell does not explicitly teach the claimed "to receive an enhanced image 
from a third sensor configured to provide an image of the out-the-window scenery in low- 
visibility conditions". Bernier teaches the claimed limitation by teaching of "Typical sensors 
used with the system are low-light video cameras, long-wave 'infrared sensors, and millimeter 
wave radar, to name a few" (paragraph [0050]). It would have been obvious to one of ordinary 
skill in the art to use the claimed feature in order to provide better navigation capabilities in the 
visualization system. 

As per claim 9, Guell does not explicitly teach the claimed limitation. Bernier teaches 
the claimed limitation by teaching of "For example, given a visible and an infrared image 
covering similar fields of view, the images can be combined at the pixel level, where priority can 
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be given to the infrared image based upon its pixel intensity" (paragraph [0017]) where the 
combining is the fusing. It would have been obvious to one of ordinary skill in the art to use the 
claimed feature of Bernier. Bernier teaches one advantage to combining the visible and infrared 
image features of Bernier with the system of Guell and NASA by teaching of "The images are 
then summed together in this intensity percentage ratio, thereby providing the user with the 
benefit of both images" (paragraph [0017]). Thus, the pilot using the imaging system of Guell 
and NASA would benefit from receiving a blended and effectively combined image display, 
because a blended image provides information of two sources simultaneously to aid with 
accurate imaging. 

As per claim 10, Guell does not explicitly teach the claimed limitation. Bernier teaches 
the claimed limitation by teaching of "Typical sensors used with the system are low-light video 
cameras, long-wave infrared sensors, and millimeter wave radar, to name a few" (paragraph 
[0050]) where these sensors can be position sensors. It would have been obvious to one of 
ordinary skill in the art to use the claimed feature in order to provide better navigation 
capabilities in the visualization system. 

As per claim 11, Guell does not explicitly teach the claimed "utilize data from off-board 
data sources regarding the objects". Bernier teaches the claimed limitation by teaching of "the 
central processor receives the output of a synthetic vision system that generates 3D terrain and 
graphics from a navigation database" (paragraph [0101]) where this database would contain off- 
broad 3D terrain data and off-broad navigation data since the aircraft would have limited 
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capabilities of obtaining ground data through on-board ground sensors if at high altitudes. It 
would have been obvious to one of ordinary skill in the art to use the claimed feature in order to 
provide better navigation capabilities in the visualization system. 

As per claim 13, Guell does not explicitly teach the claimed " wherein the third sensor is a 
RADAR". Bernier teaches the claimed limitation by teaching of 'Typical sensors used with the 
system are low-light video cameras, long-wave 'infrared sensors, and millimeter wave radar, to 
name a few" (paragraph [0050]). It would have been obvious to one of ordinary skill in the art to 
use the claimed feature in order to provide better navigation capabilities to the visualization 
system. 

As per claim 14, Guell does not explicitly teach the claimed "wherein the third sensor is a 
FLIR sensor". Bernier teaches the claimed limitation by teaching of "Typical sensors used with 
the system are low-light video cameras, long-wave 'infrared sensors, and millimeter wave radar, 
to name a few" (paragraph [0050]), where the long-wave infrared sensors perform a similar 
function to the FLIR. It would have been obvious to one of ordinary skill in the art to use the 
claimed feature in order to provide better navigation capabilities to the visualization system. 

3. Claims 16-19, 24-27, 33, 37, 38, and 46 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Guell in view of NASA in further view of Sankrithi et al. (US Patent 
6,405,975, herein referred to as "Sankrithi"). 
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As per claims 16, 24, and 37, the reasons and rationale for the rejection of claim 1 are 
incorporated herein in regards to the claimed receiving of images from sensors, the claimed 
fusing the images, and the claimed transforming the fused images to viewpoints. 

Guell does not explicitly teach the claimed "output the first operator viewpoint image ... 
wherein the display device are positioned to provide to provide the portion of a desired out-the- 
window visual scene in combination with a cockpit window that provides another portion of the 
desired out-the-window visual scene, further wherein the images on the first and second display 
devices are aligned together with the real-world view out the window view to meet operation 
field-of-view requirements". Sankrithi teaches the claimed limitation in figure 6A where the first 
display device (272L) and the second display device (272R) are shown each displaying a real- 
world view. Further, these display devices are provided in combination with the real-world 
scene from the cockpit window, where the cockpit window would be above the two display 
devices. Further, figure 6B shows that the display devices can be aligned directly next to the 
cockpit window. In this instance, the examiner is interpreting the operational field-of-view 
requirements to be front view through the window and the real-view left and right perspective 
views. 

It would have been obvious to one of ordinary skill in the art at the time of invention to 
combine Sankrithi with the combinable system of Guell and NASA. One advantage of the 
combination is provided by Sankrithi, which teaches of aiding in maneuvering airplanes (col 1, 
line 66 - col 2, line 5). 
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As per claims 17 and 18, Guell does not explicitly teach the claimed "detecting objects in 
the first fused image from the first type of sensor 11 and does not explicitly teach the claimed 
"combining the first fused image with symbols representing the objects". NASA teaches the 
claimed limitations by teaching of "Object Detection: Data was gathered to help develop 
methods for detecting other aircraft" (third item in bulleted list under section "XVS Flight Test 
Series 11"). NASA teaches of this object detection being used with the first fused image from 
the first type of sensor because it differentiates "Object Detection" from "Object Detection via 
Radar" (object detection through the second fused image) in describing data collection methods 
in the bulleted list under section "XVS Flight Test Series II". It would have been obvious to use 
the claimed feature in order to provide the pilot with better navigation data through computer- 
generated symbols. 

As per claim 19, Guell teaches the claimed "transforming the first operator viewpoint 
image and the second operator viewpoint image to conform to the out-the-window visual scene" 
by teaching of "In essence, the actual outside world view can encompass a very large area that 
can go beyond what is visually allowed by the cockpit windows, providing unobstructed views in 
any desired direction (see figure 3)" (pg. 33, last sentence of 1st column, and top of second 
column). 

As per claims 25, 26, and 27, Guell does not explicitly teach the claimed "display 
processor is further operable to combine the viewpoint image with symbols, wherein the symbols 
represent information regarding the operational state of the device and the moving objects 
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detected in the images", does not explicitly teach the claimed "display processor is further 
operable to detect moving objects in the first sensor image", and does not explicitly teach the 
claimed "display processor is further operable to generate symbols representing moving objects 
in the sensor image and the operational state of the device". NASA teaches the claimed . 
limitations by teaching of tracking objects in the section titled "XVS Flight Test Series 111" and 
by teaching of "pilot's ability to control and land the aircraft relying only on sensors and 
computer-generated images (including various symbols) on the XVS display" (3rd paragraph 
under the section titled "XVS Flight Test Series I"). NASA teaches of an advantage of 
displaying symbols of moving objects used in the XVS system by teaching of "The envisioned 
external Vision System (XVS) would guide pilots to an airport, warn them of other aircraft near 
their flight path, and provide additional visual aides for airport approaches, landings and 
takeoffsl" (2nd paragraph in article). NASA teaches of displaying symbols (a feature of the 
XVS system) that would warn pilots of other aircraft near their flight path. It would have been 
obvious to use the claimed features in order to provide the pilot with better navigation data 
through computer-generated symbols. 

As per claim 33, Guell teaches the claimed "sensor is a video camera" by teaching of 
"video images from the cameras are digitized, and electronically combined in the Electronic 
Imaging System (EIS)" (pg. 33, first full paragraph in first column). 

As per claim 38, this claim has limitations that follow those within claim 17 in terms of 
functionality, and thus are subject to the same reasons for rejection. 
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As per claim 46, Guell teaches the claimed limitations by teaching of "specific 
mission/operational needs" (pg. 33, top of 1st col) and by teaching of a "Control Panel" (figure 
1). 

4. Claims 20-23, 28-32, 34-36, 39, 41-45, and 47 are rejected under 35 U.S.C. 103(a) as 
being unpatentable over Guell in view of NASA in further view of Sankrithi in further view of 
Bernier et al. (US Pub 2004/0169663, herein referred to as "Bernier"). 

As per claims 20-23 and 41-45, Guell teaches the claimed "transforming the second fused 
image to the first operator viewpoint and to the second operator viewpoint" by teaching of "Each 
pilot has an independent view, which is controlled by their head position, while utilizing the 
same sensors that are static and fixed to the aircraft structure" (pg. 32, middle of first paragraph 
in first column) where the each pilot's independent view is an operator viewpoint. 

Guell teaches the claimed "providing portions of the transformed image with data from a 
terrain map database" by teaching of a "Moving Map / RTIC" displayed as a portion of the 
transformed image in pg. 34 figure 6. 

Guell does not explicitly teach the claimed fusing the first and second images "with an 
enhanced image of a portion of the out-the-window scenery from at least one of the group of a 
RADAR sensor and a FLIR sensor, to generate a second fused image". Bernier teaches the 
claimed limitation by teaching of "Typical sensors used with the system are low-light video 
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cameras, longwave infrared sensors, and millimeter wave radar, to name a few" (paragraph 
[0050]) where the long-wave infrared sensors perform a similar to function to the FLIR. 

It would have been obvious to one of ordinary skill in the art at the time of invention to 
combine Bernier with the combinable system of Guell, NASA, and Sankrithi. One advantage of 
the combination is provided by Bernier, which teaches of "the system of the present invention 
may include two different types of sources ... the sources provide different images of the same 
field of view; each source having associated advantages and disadvantages. For example, one 
source could be a video camera . . . and the other source may be an infrared sensor that provides 
images based on heat sensing" (paragraph [0016]). The pilot of the system of Guell would 
benefit from image sources, which provide image data in a variety of environmental conditions 
(i.e. low-visibility conditions) such as the infrared sources because it would help the pilot see 
more information as to the surroundings of the aircraft. 

As per claim 28, Guell does not explicitly teach the claimed limitation. NASA teaches of 
computer-generated symbols on the display (2 nd paragraph under section titled C XVS Flight Test 
Series I"). Bernier teaches the claimed weather hazards "Visual hindrances may be due to bad 
weather, such as fog, snow, or rain, or they may be due to the time of day, such as night, dawn, 
or dusk. Further, some visual hindrances are due to the field of view limitations of the aircraft 
itself. Many aircraft cockpits have a field of view that is typically limited to a forward facing 
area that does not provide the pilot with adequate visualization to the sides and rear of the 
aircraft" (paragraph [0005]). It would have been obvious to one of ordinary skill in the art to use 
the claimed feature as taught by Bernier with Guell. Bernier teaches one advantage to the 
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combining visible and computer generated image features by teaching of "The images are then 
summed together in this intensity percentage ratio, thereby providing the user with the benefit of 
both images" (paragraph [0017]). 

As per claim 29, Guell does not explicitly teach the claimed limitation. Bernier teaches 
the claimed limitation by teaching of "Typical sensors used with the system are low-light video 
cameras, long- wave 'infrared sensors, and millimeter wave radar, to name a few" (paragraph 
[0050]). It would have been obvious to one of ordinary skill in the art to use the claimed feature 
in order to provide better navigation capabilities in the visualization system. 

As per claim 30, Guell does not explicitly teach the claimed limitation. Bernier teaches 
the claimed limitation by teaching of "For example, given a visible and an infrared image 
covering similar fields of view, the images can be combined at the pixel level, where priority can 
be given to the infrared image based upon its pixel intensity" (paragraph [0017]) where the 
combining is the fusing. It would have been obvious to one of ordinary skill in the art to use the 
claimed feature of Bernier with Guell. Bernier teaches one advantage to combining the visible 
and infrared image features of Bernier with the system of Guell and NASA in paragraph [0017]. 

As per claim 31, Guell does not explicitly teach the claimed limitation. Bernier teaches 
the claimed limitation by teaching of "Typical sensors used with the system are low-light video 
cameras, long-wave infrared sensors, and millimeter wave radar, to name a few" (paragraph 
[0050]) where these sensors can be position sensors. It would have been obvious to one of 
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ordinary skill in the art to use the claimed feature in order to provide better navigation 
capabilities in the visualization system. 

As per claim 32, Guell does not explicitly teach the claimed "utilize data from off-board 
data sources regarding the objects" Bernier teaches the claimed limitation by teaching of "the 
central processor receives the output of a synthetic vision system that generates 3D terrain and 
graphics from a navigation database" (paragraph [0101]) where this database would contain off- 
broad 3D terrain data and off-broad navigation data since the aircraft would have limited 
capabilities of obtaining ground data through on-board ground sensors if at high altitudes. It 
would have been obvious to one of ordinary skill in the art to use the claimed feature in order to 
provide better navigation capabilities in the visualization system. 

As per claim 34, Guell does not explicitly teach the claimed "wherein the second sensor 
is a RADAR". Bernier teaches the claimed limitation by teaching of "Typical sensors used with 
the system are low-light video cameras, long-wave 'infrared sensors, and millimeter wave radar, 
to name a few" (paragraph [0050]). It would have been obvious to one of ordinary skill in the art 
to use the claimed feature in order to aid the pilot in navigating in low- visibility light. 

As per claim 35, Guell does not explicitly teach the claimed "wherein the second sensor 
is a FLIR sensor". Bernier teaches the claimed limitation by teaching of "Typical sensors used 
with the system are low-light video cameras, long- wave 'infrared sensors, and millimeter wave 
radar, to name a few" (paragraph [0050]), where the long-wave infrared sensors perform a 
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similar function to the FLIR. It would have been obvious to one of ordinary skill in the art to use 
the claimed feature in order to aid the pilot in navigating in low- visibility light. 

_ As per claim 39, Guell does not explicitly teach the claimed limitation. Bemier teaches 
the claimed "primary flight information" by teaching of "For example, the aircrafts HSI and AD1 
displays, altimeters, airspeed, etc, could be displayed on the display as a tile or fused with an 
image to provide an integrated view ... while also viewing the environment surrounding the 
aircraft" (paragraph [0102]). It would have been obvious to one of ordinary skill in the art to use 
the claimed feature in order to aid the pilot by providing important navigation data. 

As per claims 36, and 47, Guell does not teach the claimed limitations. Bernier teaches 
the claimed limitation by teaching of in figure 16 a common window, 124 is generate from two 
separate image sources 122a and 122b (windows of information). Bernier in figure 18 shows 
that the common window can have mutually exclusive (not overlapping) windows of 
information, 134 and 136 and that the common window is customizable where the added 
windows of information 134 and 136 are customized added features. It would have been obvious 
to one of ordinary skill in the art to use the claimed feature in order to aid the pilot by providing 
important navigation data through a convenient user interface. 

Response to Arguments 

5. Applicant's arguments with respect to the claims have been considered but are moot in 
view of the new ground(s) of rejection. 
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Conclusion 



Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Daniel F. Hajnik whose telephone number is (571) 272-7642. 
The examiner can normally be reached on Mon-Fri (8:30A-5:00P). 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Ulka J. Chauhan can be reached on (571) 272-7782. The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 
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